fix(lambda-python): root-owned cache items not cleaned up after install #22512

varju · 2022-10-15T01:05:50Z

This is another attempt at fixing #22012. #22398 intended to fix it, but the issue still occurs when building on arm64 machines with both compatible_runtimes=[lambdas.Runtime.PYTHON_3_9] and compatible_architectures=[lambdas.Architecture.ARM_64]

The core problem is that the build leaves root-owned files under /tmp/pip-cache, which then cause permission problems when subsequent layers calling pip install are unable to create temporary directories. Example error:

WARNING: Building wheel for alembic-utils failed: [Errno 13] Permission denied: '/tmp/pip-cache/wheels/5c'

Expanding on the idea within #22398, this PR adds one more step to the chain of operations, removing any temporary files from the cache that are no longer needed (because the packages are now installed), and ensuring that subsequent layers are able to create temporary files inside the cache directories.

Comparing the contents of the Docker image before and after this change...

Before:

$ ls -aFl /tmp/*cache
/tmp/pip-cache:
total 16
drwxrwxrwx  4 root root 4096 Oct 15 00:57 ./
drwxrwxrwt  1 root root 4096 Oct 15 00:58 ../
drwxr-xr-x 18 root root 4096 Oct 15 00:57 http/
drwxr-xr-x  2 root root 4096 Oct 15 00:57 selfcheck/

/tmp/poetry-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 00:57 ./
drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../

After:

$ ls -aFl /tmp/*cache
/tmp/pip-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 01:00 ./
drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../

/tmp/poetry-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 00:59 ./
drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../

All Submissions:

Have you followed the guidelines in our Contributing guide?

Adding new Unconventional Dependencies:

This PR adds new unconventional dependencies following the process described here

New Features

Have you added the new feature to an integration test?
- Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

gitpod-io · 2022-10-15T01:05:58Z

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.

varju · 2022-10-17T22:39:14Z

packages/@aws-cdk/aws-lambda-python/test/packaging.test.ts

@@ -1,31 +1,31 @@
 import * as path from 'path';
 import { Packaging } from '../lib/packaging';

-test('Packging with no dependencies', () => {
+test('Packaging with no dependencies', () => {


Fixing a typo unrelated to the current problem. I can revert this if it's not okay to mix unrelated changes.

I'm never upset about typo fixes :)

TheRealAmazonKendra · 2022-10-19T00:26:12Z

packages/@aws-cdk/aws-lambda-python/test/packaging.test.ts

@@ -1,31 +1,31 @@
 import * as path from 'path';
 import { Packaging } from '../lib/packaging';

-test('Packging with no dependencies', () => {
+test('Packaging with no dependencies', () => {


I'm never upset about typo fixes :)

mergify · 2022-10-19T00:32:22Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

Pull request has been modified.

aws-cdk-automation · 2022-10-19T13:54:02Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 3b675d7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2022-10-19T13:54:25Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

…ll (aws#22512) This is another attempt at fixing aws#22012. aws#22398 intended to fix it, but the issue still occurs when building on arm64 machines with both `compatible_runtimes=[lambdas.Runtime.PYTHON_3_9]` and `compatible_architectures=[lambdas.Architecture.ARM_64]` The core problem is that the build leaves root-owned files under `/tmp/pip-cache`, which then cause permission problems when subsequent layers calling `pip install` are unable to create temporary directories. Example error: ``` WARNING: Building wheel for alembic-utils failed: [Errno 13] Permission denied: '/tmp/pip-cache/wheels/5c' ``` Expanding on the idea within aws#22398, this PR adds one more step to the chain of operations, removing any temporary files from the cache that are no longer needed (because the packages are now installed), and ensuring that subsequent layers are able to create temporary files inside the cache directories. Comparing the contents of the Docker image before and after this change... Before: ``` $ ls -aFl /tmp/*cache /tmp/pip-cache: total 16 drwxrwxrwx 4 root root 4096 Oct 15 00:57 ./ drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../ drwxr-xr-x 18 root root 4096 Oct 15 00:57 http/ drwxr-xr-x 2 root root 4096 Oct 15 00:57 selfcheck/ /tmp/poetry-cache: total 8 drwxrwxrwx 2 root root 4096 Oct 15 00:57 ./ drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../ ``` After: ``` $ ls -aFl /tmp/*cache /tmp/pip-cache: total 8 drwxrwxrwx 2 root root 4096 Oct 15 01:00 ./ drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../ /tmp/poetry-cache: total 8 drwxrwxrwx 2 root root 4096 Oct 15 00:59 ./ drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../ ``` ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK p2 labels Oct 15, 2022

aws-cdk-automation requested a review from a team October 15, 2022 01:06

aws-cdk-automation previously requested changes Oct 15, 2022

View reviewed changes

varju changed the title ~~Clean up root-owned cache items~~ fix(lambda-python): Clean up root-owned cache items Oct 15, 2022

varju changed the title ~~fix(lambda-python): Clean up root-owned cache items~~ fix(lambda-python): clean up root-owned cache items Oct 15, 2022

varju added 4 commits October 17, 2022 15:36

Fix typo

a8eacb5

Test demonstrating failure

b43ce4e

Clean up root-owned cache items

3e9b603

Update test snapshot now that it passes

4664d9d

varju force-pushed the 22012-arm64-docker-python-layer branch from e1f074b to 4664d9d Compare October 17, 2022 22:37

varju commented Oct 17, 2022

View reviewed changes

varju marked this pull request as ready for review October 17, 2022 23:56

TheRealAmazonKendra changed the title ~~fix(lambda-python): clean up root-owned cache items~~ fix(lambda-python): root-owned cache items not cleaned up after install Oct 19, 2022

TheRealAmazonKendra previously approved these changes Oct 19, 2022

View reviewed changes

Merge branch 'main' into 22012-arm64-docker-python-layer

3b675d7

TheRealAmazonKendra approved these changes Oct 19, 2022

View reviewed changes

mergify bot merged commit 5ef65e0 into aws:main Oct 19, 2022

varju deleted the 22012-arm64-docker-python-layer branch October 19, 2022 16:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(lambda-python): root-owned cache items not cleaned up after install #22512

fix(lambda-python): root-owned cache items not cleaned up after install #22512

varju commented Oct 15, 2022 •

edited

Loading

gitpod-io bot commented Oct 15, 2022

aws-cdk-automation left a comment

varju Oct 17, 2022

TheRealAmazonKendra Oct 19, 2022

TheRealAmazonKendra Oct 19, 2022

mergify bot commented Oct 19, 2022

aws-cdk-automation commented Oct 19, 2022

mergify bot commented Oct 19, 2022

fix(lambda-python): root-owned cache items not cleaned up after install #22512

fix(lambda-python): root-owned cache items not cleaned up after install #22512

Conversation

varju commented Oct 15, 2022 • edited Loading

All Submissions:

Adding new Unconventional Dependencies:

New Features

gitpod-io bot commented Oct 15, 2022

aws-cdk-automation left a comment

Choose a reason for hiding this comment

varju Oct 17, 2022

Choose a reason for hiding this comment

TheRealAmazonKendra Oct 19, 2022

Choose a reason for hiding this comment

TheRealAmazonKendra Oct 19, 2022

Choose a reason for hiding this comment

mergify bot commented Oct 19, 2022

aws-cdk-automation commented Oct 19, 2022

AWS CodeBuild CI Report

mergify bot commented Oct 19, 2022

varju commented Oct 15, 2022 •

edited

Loading